Image processing apparatus, image processing method, and computer-readable recording medium

ABSTRACT

An image processing apparatus includes: a region-of-interest setting unit configured to set a region of interest in an image; a linear convex region extracting unit configured to extract, from the region of interest, a linear region having a predetermined number or more of continuously-arranged pixels whose pixel values are higher than pixel values of neighboring pixels; an intra-region curvature feature data computing unit configured to compute curvature feature data based on curvatures of one or more arcs along the linear region; and an abnormality determining unit configured to determine whether there is an abnormal portion in the region of interest, based on a distribution of the curvature feature data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of PCT international application Ser.No. PCT/JP2013/084137 filed on Dec. 19, 2013 which designates the UnitedStates, incorporated herein by reference.

BACKGROUND

1. Technical Field

The disclosure relates to an image processing apparatus, an imageprocessing method, and a computer-readable recording medium fordetermining whether there is an abnormal portion in an image obtained byimaging an in-vivo lumen.

2. Related Art

For image processing performed on an intraluminal image (hereinafter,also simply referred to an image) which is acquired by observing themucosa of the large intestine with a magnification endoscope, forexample, Japanese Laid-open Patent Publication No. 2007-236956 disclosesa technique for classifying a pit pattern of a mucosal surface (called alarge intestine pit pattern) into a plurality of types. Specifically,Gabor filters with m frequencies and k phase orientations are applied toeach pixel in a region of interest which is set in an image, by whichm×k-dimensional feature vectors are computed, and the mean or varianceof the feature vectors is computed. Then, a large intestine pit patternis classified based on the mean or variance of the feature vectors, andit is determined whether the large intestine pit pattern is abnormal.

SUMMARY

In some embodiments, an image processing apparatus includes: aregion-of-interest setting unit configured to set a region of interestin an image; a linear convex region extracting unit configured toextract, from the region of interest, a linear region having apredetermined number or more of continuously-arranged pixels whose pixelvalues are higher than pixel values of neighboring pixels; anintra-region curvature feature data computing unit configured to computecurvature feature data based on curvatures of one or more arcs along thelinear region; and an abnormality determining unit configured todetermine whether there is an abnormal portion in the region ofinterest, based on a distribution of the curvature feature data.

In some embodiments, an image processing method includes: setting aregion of interest in an image; extracting, from the region of interest,a linear region having a predetermined number or more ofcontinuously-arranged pixels whose pixel values are higher than pixelvalues of neighboring pixels; computing curvature feature data based oncurvatures of one or more arcs along the linear region; and determiningwhether there is an abnormal portion in the region of interest, based ona distribution of the curvature feature data.

In some embodiments, a non-transitory computer-readable recording mediumwith an executable program stored thereon is provided. The programinstructs a processor to execute: setting a region of interest in animage; extracting, from the region of interest, a linear region having apredetermined number or more of continuously-arranged pixels whose pixelvalues are higher than pixel values of neighboring pixels; computingcurvature feature data based on curvatures of one or more arcs along thelinear region; and determining whether there is an abnormal portion inthe region of interest, based on a distribution of the curvature featuredata.

The above and other features, advantages and technical and industrialsignificance of this invention will be better understood by reading thefollowing detailed description of presently preferred embodiments of theinvention, when considered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of an imageprocessing apparatus according to a first embodiment of the presentinvention;

FIG. 2 is a flowchart illustrating the operation of the image processingapparatus illustrated in FIG. 1;

FIG. 3 is a flowchart illustrating the details of processes performed bya region-of-interest setting unit illustrated in FIG. 1;

FIG. 4 is a flowchart illustrating the details of processes performed bya mucosal region extracting unit illustrated in FIG. 1;

FIG. 5 is an example of creation of a histogram showing the frequency ofhue mean values;

FIG. 6 is a schematic diagram describing a clustering method fordividing a histogram into a plurality of clusters using the valleys ofthe histogram as boundaries;

FIG. 7 is a graph illustrating results obtained by clustering thehistogram illustrated in FIG. 5;

FIG. 8 is a flowchart illustrating the details of processes performed bya linear convex region extracting unit illustrated in FIG. 1;

FIG. 9 is a schematic diagram describing a top-hat transform;

FIG. 10 is a diagram illustrating patterns used to search in an image ina thinning filtering process;

FIG. 11 is a flowchart illustrating the details of processes performedby an intra-region curvature feature data computing unit illustrated inFIG. 1;

FIG. 12 is a flowchart illustrating the details of processes performedby a size feature data computing unit illustrated in FIG. 1;

FIG. 13 is a schematic diagram illustrating sections of linear regions;

FIG. 14 is a schematic diagram describing a method of computing acurvature of an arc;

FIG. 15 is a schematic diagram describing a process performed by acurvature computing unit illustrated in FIG. 1;

FIG. 16A is a schematic diagram illustrating a normal villus model;

FIG. 16B is a schematic diagram illustrating convex regionscorresponding to the normal villus model;

FIG. 17A is a schematic diagram illustrating an abnormal villus model;

FIG. 17B is a schematic diagram illustrating linear regionscorresponding to abnormal villi;

FIG. 17C is a graph illustrating a distribution of curvatures of arcsalong the linear regions corresponding to abnormal villi;

FIG. 18A is a schematic diagram illustrating a bubble model;

FIG. 18B is a schematic diagram illustrating linear regionscorresponding to a bubble region;

FIG. 18C is a graph illustrating a distribution of curvatures of arcsalong the linear regions corresponding to a bubble region;

FIG. 19 is a schematic diagram illustrating a frequency distributionconsisting of two axes, arc curvature and distance information;

FIG. 20 is a block diagram illustrating an image processing apparatusaccording to a modification 1-2 of the first embodiment;

FIG. 21 is a flowchart illustrating the details of processes performedby a linear convex region extracting unit illustrated in FIG. 20;

FIG. 22 is a schematic diagram illustrating patterns used in a Hilditchthinning algorithm;

FIG. 23 is a schematic diagram illustrating patterns used in theHilditch thinning algorithm;

FIG. 24 is a block diagram illustrating a configuration of an imageprocessing apparatus according to a second embodiment of the presentinvention;

FIG. 25 is a flowchart illustrating processes performed by anintra-region curvature feature data computing unit illustrated in FIG.24;

FIG. 26 is a flowchart illustrating processes performed by a shapefeature data computing unit illustrated in FIG. 24;

FIG. 27A is a schematic diagram illustrating an abnormal villus model;

FIG. 27B is a schematic diagram illustrating the curvatures of an arcalong a linear region corresponding to an abnormal villus;

FIG. 27C is a graph illustrating standard deviations of curvaturescomputed for linear regions corresponding to abnormal villi;

FIG. 28A is a schematic diagram illustrating a bubble model;

FIG. 28B is a schematic diagram illustrating curvatures of arcs along alinear region corresponding to a bubble region;

FIG. 28C is a graph illustrating the standard deviations of curvaturescomputed for linear regions corresponding to a bubble region;

FIG. 29 is a block diagram illustrating a configuration of an imageprocessing apparatus according to a third embodiment of the presentinvention;

FIG. 30 is a flowchart illustrating processes performed by anintra-region curvature feature data computing unit illustrated in FIG.29;

FIG. 31 is a flowchart illustrating processes performed by a directionfeature data computing unit illustrated in FIG. 29;

FIG. 32A is a schematic diagram illustrating an abnormal villus model;

FIG. 32B is a schematic diagram illustrating linear regionscorresponding to abnormal villi;

FIG. 32C is a schematic diagram illustrating the curvature centraldirections of arcs along the linear regions corresponding to abnormalvilli;

FIG. 32D is a graph illustrating the frequency of curvature centraldirections for abnormal villi for each gradient direction;

FIG. 33A is a schematic diagram illustrating a bubble model;

FIG. 33B is a schematic diagram illustrating linear regionscorresponding to a bubble region;

FIG. 33C is a schematic diagram illustrating the curvature centraldirections of arcs along the linear regions corresponding to a bubbleregion;

FIG. 33D is a graph illustrating the frequency of curvature centraldirections for a bubble region for each gradient direction;

FIG. 34 is a block diagram illustrating a configuration of an imageprocessing apparatus according to a fourth embodiment of the presentinvention;

FIG. 35 is a flowchart illustrating processes performed by aregion-of-interest setting unit illustrated in FIG. 34; and

FIG. 36 is a flowchart illustrating the operation of an image processingapparatus of a modification 4-1 of the fourth embodiment.

DETAILED DESCRIPTION

Image processing apparatuses, image processing methods, and imageprocessing programs according to some embodiments of the presentinvention will be described below with reference to the drawings. Thepresent invention is not limited to these embodiments. The samereference signs are used to designate the same elements throughout thedrawings.

First Embodiment

FIG. 1 is a block diagram illustrating an image processing apparatusaccording to a first embodiment of the present invention. An imageprocessing apparatus 1 according to the first embodiment is, as anexample, an apparatus that performs image processing for determiningwhether there is a pathologically changed portion (abnormal portion)where a villus which is a microstructure of a mucosal surface isswollen, on an intraluminal image (hereinafter, also simply referred toas an image) which is acquired by capturing an in-vivo lumen by acapsule endoscope. The intraluminal image is normally a color imagehaving a predetermined (e.g., 256 shades) pixel level (pixel value) forR (red), G (green), and B (blue) wavelength components (colorcomponents) in each pixel position.

As illustrated in FIG. 1, the image processing apparatus 1 includes acontrol unit 10 that controls the entire operation of the imageprocessing apparatus 1; an image acquiring unit 20 that acquires imagedata for an image captured by an endoscope; an input unit 30 thataccepts input signals to be inputted from an external source; a displayunit 40 that performs various types of display; a recording unit 50 thatstores the image data acquired by the image acquiring unit 20 andvarious programs; and a calculating unit 100 that performs predeterminedimage processing on the image data.

The control unit 10 is implemented by hardware such as a CPU. By readingvarious types of programs recorded in the recording unit 50, the controlunit 10 performs, for example, instructions and data transfer to theunits composing the image processing apparatus 1, according to imagedata inputted from the image acquiring unit 20, an operation signalinputted from the input unit 30, etc., and thereby performs overallcontrol of the entire operation of the image processing apparatus 1.

The image acquiring unit 20 is composed as appropriate, according to themode of a system including an endoscope. For example, when a portablerecording medium is used to pass image data with a capsule endoscope,the image acquiring unit 20 is composed of a reader device that allowsthe recording medium to be removably placed therein and that reads imagedata of a recorded image. In addition, when a server that saves imagedata of images captured by the endoscope is set up, the image acquiringunit 20 is composed of a communication device or the like which isconnected to the server, and acquires image data by performing datacommunication with the server. Alternatively, the image acquiring unit20 may be composed of an interface device or the like that accepts asinput an image signal from the endoscope through a cable.

The input unit 30 is implemented by input devices, e.g., a keyboard, amouse, a touch panel, and various types of switches, and outputs anaccepted input signal to the control unit 10.

The display unit 40 is implemented by a display device such as an LCD oran EL display, and displays various types of screens including anintraluminal image, under control of the control unit 10.

The recording unit 50 is implemented by, for example, various types ofIC memories such as a ROM and a RAM, e.g., updatable and recordableflash memories, a hard disk which is built in or connected by a datacommunication terminal, or an information recording device such as aCD-ROM and a reading device therefor. The recording unit 50 stores aprogram that causes the image processing apparatus 1 to operate andcauses the image processing apparatus 1 to perform various functions,data to be used during the execution of the program, etc., in additionto image data acquired by the image acquiring unit 20. Specifically, therecording unit 50 stores an image processing program 51 for determiningwhether there is an abnormal portion of a villus on a mucosal surface,various information to be used during the execution of the program, etc.

The calculating unit 100 is implemented by hardware such as a CPU. Byreading the image processing program 51, the calculating unit 100performs image processing on an intraluminal image, and performs variouscalculation processes for determining whether there is an abnormalportion of a villus on a mucosal surface.

Next, a configuration of the calculating unit 100 will be described.

As illustrated in FIG. 1, the calculating unit 100 includes aregion-of-interest setting unit 110 that sets a region of interest in aprocessing target intraluminal image; a linear convex region extractingunit 120 that extracts regions in the form of a line (hereinafter, alsoreferred to as linear regions) each having a predetermined number ormore of continuously-arranged pixels whose pixel values are higher thanthose of their neighboring pixels; an intra-region curvature featuredata computing unit 130 that computes a distribution of feature databased on the curvatures of one or more arcs along each of the linearregions (hereinafter, also referred to as curvature feature data); andan abnormality determining unit 140 that determines whether there is anabnormal portion in the region of interest, based on the curvaturefeature data.

Of them, the region-of-interest setting unit 110 includes a mucosalregion extracting unit 111 that extracts a mucosal region by excludingregions other than mucosa such as residues and dark portions from theimage, and sets the extracted mucosal region as a region of interest.

The linear convex region extracting unit 120 includes a convex shapehigh-frequency component computing unit 121, an isolated-point excludingunit 122, and a thinning unit 123.

The convex shape high-frequency component computing unit 121 computesthe strengths of components whose spatial frequencies are greater thanor equal to a predetermined value (hereinafter, referred to ashigh-frequency components), in a pixel region with higher pixel valuesthan its neighboring pixels. Note that in the following the pixel regionwith higher pixel values than its neighboring pixels is also referred toas a convex-shaped region or simply a convex shape, and a pixel regionthat has the convex shape and has high-frequency components whosestrengths are greater than or equal to the predetermined value is alsoreferred to as a convex-shaped high-frequency region. In addition, thestrengths of high-frequency components in the convex-shapedhigh-frequency region are also referred to as projection shape'shigh-frequency components.

The isolated-point excluding unit 122 excludes isolated points as normalvilli, based on the projection shape's high-frequency componentscomputed by the convex shape high-frequency component computing unit121. Here, the isolated point refers to a region in the convex-shapedhigh-frequency region that has a smaller number of consecutive pixelsthan a predetermined threshold value in all circumferential directionsof the convex-shaped high-frequency region.

The thinning unit 123 thins the convex-shaped high-frequency region.

The intra-region curvature feature data computing unit 130 includes asize feature data computing unit 131 that computes the curvatures of oneor more arcs along a linear region and distance informationcorresponding to imaging distances to the arcs, as arc size featuredata; and a frequency distribution creating unit 132 that creates afrequency distribution of the arc size feature data. Of them, the sizefeature data computing unit 131 includes a curvature computing unit 131a that computes the curvatures of one or more arcs from a linear region;a curvature representative value computing unit 131 b that computes arepresentative value from the computed curvatures of one or more arcs;and a distance information computing unit 131 c that computes distanceinformation from an intraluminal image imaging position (i.e., theposition of the capsule endoscope) to the convex region that forms alinear shape.

Next, the operation of the image processing apparatus 1 will bedescribed. FIG. 2 is a flowchart illustrating the operation of the imageprocessing apparatus 1.

First, at step S10, the calculating unit 100 reads image data recordedin the recording unit 50 and thereby acquires a processing targetintraluminal image.

At subsequent step S20, the region-of-interest setting unit 110 sets aregion of interest in the intraluminal image. FIG. 3 is a flowchartillustrating the details of processes performed by theregion-of-interest setting unit 110.

At step S201, the mucosal region extracting unit 111 divides theintraluminal image into a plurality of small regions, based on the edgestrengths of the respective pixels in the intraluminal image. Theprocess of dividing the intraluminal image into a plurality of smallregions will be described in detail with reference to FIG. 4. First, atstep S2011, the mucosal region extracting unit 111 creates a G-componentimage, using G components included in the pixel values of the respectivepixels in the intraluminal image. The reason that the G components areused here is that the G components are close to a blood absorption bandand thus provides the best representation of the structures of objectssuch as mucosal bumps and residue boundaries. Note that although in thefirst embodiment the G components are used, instead, for example, othercolor components, or a luminance value, a color difference (YCbCrconversion), hue, saturation, intensity (HSI conversion), or a colorratio, which is obtained by converting a pixel value (R, G, and Bcomponents) may be used.

At step S2012, the mucosal region extracting unit 111 computes edgestrengths by performing a filtering process on the G-component image,and creates an edge image that uses the edge strengths as the pixelvalues of the respective pixels. For the filtering process for computingedge strengths, a first derivative filter such as a Prewitt filter or aSobel filter, a second derivative filter such as a Laplacian filter or aLOG (Laplacian of Gaussian) filter, or the like, is used (reference:Computer Graphic Arts Society, “Digital Image Processing”, pp. 114-121).

The mucosal region extracting unit 111 further performs region divisionbased on the edge strengths of the edge image. As an example of a regiondivision technique, in the first embodiment, a method disclosed in WO2006/080239 A is applied. Specifically, at step S2013, a smoothingprocess is performed on the edge image to remove noise. At subsequentstep S2014, a direction in which the maximum gradient of edge strengthis obtained (maximum gradient direction) is acquired for each pixelposition in the edge image. Then, at step S2015, a search is performedin the edge image from each pixel (starting pixel) in the maximumgradient direction, and a pixel position is acquired where the pixelvalue (edge strength) reaches an extreme value. Furthermore, at stepS2016, a labeled image is created in which starting pixels that obtainan extreme value at the same pixel position are labeled as one region.Here, the labeled image refers to an image that uses label numbers (1 ton, n is the number of labels) which are obtained by labeling, as thepixel values of the respective pixels. A pixel region having the samepixel value in the labeled image corresponds to each small regionobtained by performing region division on the intraluminal image.Thereafter, processing returns to the main routine.

Note that for a method of dividing the intraluminal image into regions,any publicly known method other than the above-described method may beapplied. Specifically, a watershed algorithm (reference: Luc Vincent andPierre Soille, “Watersheds in digital spaces: An efficient algorithmbased on immersion simulations”, IEEE Transactions on Pattern Analysisand Machine Intelligence, Vol. 13, No. 6, pp. 583-598, June 1991) may beused. The watershed algorithm is a method of dividing an image suchthat, when water is filled into terrain where the pixel valueinformation of an image is regarded as a height, boundaries are createdbetween water accumulated in different depressions.

In addition, in the first embodiment, region division based on the edgeis performed for the purpose of reducing the influence of isolated pixelnoise and facilitating an abnormal determination at a subsequent stageby region division along mucosal or residue boundaries, etc.; however,as another region division technique, the intraluminal image may bedivided into rectangular regions of a predetermined size. In this case,processes such as creation of an edge image and region division based onthe edge strength become unnecessary, enabling to reduce processingtime. Alternatively, without performing region division, each pixel maybe used as a processing unit in each step which will be described below.In this case, the process at step S201 can be omitted.

At step S202 subsequent to step S201, the mucosal region extracting unit111 computes a mean value of feature data for each small region which isobtained by the region division. More specifically, the mucosal regionextracting unit 111 computes a mean value of each of R, G, and Bcomponents from the pixel values of the respective pixels in the smallregion. Then, by performing HSI conversion (reference: Computer GraphicArts Society, “Digital Image Processing”, p. 64 (HSI conversion andinverse conversion)) on these mean values, a hue mean value of the smallregion is computed. The reason that the hue mean value is computed hereis that objects such as the mucosa and the contents in a lumen can bedetermined to a certain extent by color feature data such as hue, fromdifferences in the absorption characteristics of blood, bile, etc.,which are the components of the mucosa and the contents in a lumen.Basically, the hues of stomach mucosa, intestinal mucosa, and contentschange in this order from a red-based hue to a yellow-based hue.

Note that a hue mean value of each small region may be obtained bycomputing hues by performing HSI conversion on the pixel values (R, G,and B components) of the respective pixels in the small region, and thencomputing a mean value of the hues of the respective pixels.Alternatively, instead of hues, for example, a mean value of featuredata such as color differences or color ratios (G/R or B/G) may becomputed.

At step S203, the mucosal region extracting unit 111 determines, basedon the mean values of feature data, whether each small region is a darkregion. More specifically, the mucosal region extracting unit 111 firstcreates a determination result list for the small regions. Here, thedetermination result list refers to a list where the label numbers inthe labeled image obtained when the intraluminal image is divided into aplurality of small regions are associated with pieces of flaginformation for the label numbers, respectively. Note that the size ofthe determination result list corresponds to the number of labels n.Subsequently, the mucosal region extracting unit 111 initializes thedetermination result list, and assigns a mucosal region flag (0: mucosalregion) to all of the small regions. Then, using, as feature data, Gcomponents that provide the best representation of the structures ofobjects such as mucosal bumps and residue boundaries, a dark region flag(1: dark region) is assigned to a small region whose mean value of thefeature data is less than or equal to a predetermined threshold value.Note that the threshold value used at this time is determined within arange where a change in the hue mean value, etc., of the small regionmaintains linearity with respect to a change in intensity. This isbecause in the dark region the linearity breaks down due to theinfluence of noise, etc.

At step S204, the mucosal region extracting unit 111 clusters adistribution of the mean values of feature data (hue mean values) whichare computed at step S202. Here, the clustering is a technique fordividing a data distribution in a feature space into chunks calledclusters, based on the similarity between data, and can be performed byvarious publicly known techniques such as a hierarchical method and ak-means method (reference: Computer Graphic Arts Society: “Digital ImageProcessing”, pp. 231-232 (clustering)). In the first embodiment, since aprocess using a histogram of data in a hue space is performed at asubsequent stage, there is shown a procedure of clustering based on thehistogram of data, with the feature space used as a one-dimensionalspace of hue. Note that the clustering does not need to be limitedthereto, and other techniques may be used.

The process of clustering a distribution of hue mean values will bedescribed in detail below with reference to FIGS. 5 to 7. First, ahistogram with a horizontal axis being the class intervals of hue meanvalues and a vertical axis being the frequencies of hue mean valuescorresponding to each class interval is created. At this time, the huemean values are acquired from small regions that are assigned themucosal region flag (0: mucosal region) in the determination resultlist. In addition, the class intervals are set by dividing ayellow-based to red-based hue range which sufficiently includes a huedistribution range of the intraluminal image, by predetermined intervalswhich are determined in advance. FIG. 5 is an example of creation of ahistogram. Note that in FIG. 5 normalization is performed such that thesum total of frequencies is 1.

Subsequently, a distribution of the hue mean values is divided into aplurality of clusters using the valleys of the histogram as boundaries.FIG. 6 is a schematic diagram describing a clustering method. Afrequency data graph illustrated in FIG. 6 represents changes in thefrequency of the histogram by a line in a simplified manner. Inaddition, here, the class intervals represented on the horizontal axisof the histogram are represented by coordinates for convenience sake.Note that to limit the number of clustered clusters, the width of theclass intervals of the histogram may be changed or a smoothing processmay be performed on the histogram before clustering.

For such frequency data, a gradient direction for each coordinate isdetermined. Here, the gradient direction is a direction that isdetermined based on the difference between frequency data for acoordinate of interest and frequency data for coordinates adjacent tothe coordinate of interest. A direction in which the value of frequencyincreases is represented by an arrow. Note that a coordinate providedwith “extreme” is an extreme-value (maximum-value) coordinate whosefrequency is higher than any of its adjacent coordinates.

Furthermore, an extreme-value coordinate is searched for from eachcoordinate in a gradient direction. In FIG. 6, changes in a coordinatein searching for an extreme-value coordinate are represented bynumerical values of coordinates from a position of a starting coordinate(each coordinate with n=0) in a downward direction. For example, whenthe search starts with a coordinate “1”, since the gradient direction ofthe coordinate “1” is a right direction, a coordinate “2” on theimmediate right of the coordinate “1” is obtained at the first searchstep (n=1). Subsequently, since the gradient direction of the coordinate“2” is also the right direction, a coordinate “3” on the immediate rightof the coordinate “2” is obtained at the second search step (n=2).Thereafter, the search is sequentially continued in the gradientdirection, thereby eventually reaching a coordinate “5” which is anextreme-value coordinate (n=4). Next, when the search starts with thecoordinate “2”, the coordinate “5” which is an extreme-value coordinateis still eventually reached (n=3). Likewise, when the same search isperformed starting with each of all coordinates, the coordinate “5” isreached in the case of the coordinates “1” to “9” being a startingcoordinate, and a coordinate “12” is reached in the case of thecoordinates “10” to “15” being a starting coordinate.

As a result of this search, the coordinates “1” to “9” that obtain thecoordinate “5” as an extreme-value coordinate are set as a firstcluster, and the coordinates “10” to “15” that obtain the coordinate“12” as an extreme-value coordinate are set as a second cluster, bywhich the data distribution can be divided into clusters using thevalleys of the histogram as boundaries.

FIG. 7 is a graph illustrating the result of applying theabove-described clustering method to the histogram illustrated in FIG.5. In the case of FIG. 5, the hue mean values can be classified intothree clusters 1 to 3 using the valleys of the histogram as boundaries.

At step S205 subsequent to step S204, the mucosal region extracting unit111 determines unnecessary regions such as residues, based on thedistribution of feature data mean values. Here, in the lumen, ingeneral, due to the difference in absorption characteristics betweenbile and blood, unnecessary regions such as residues have a yellow-basedcolor. Hence, of the clusters obtained at step S204, a cluster whose hueleans toward yellow-based is estimated as a cluster including hue meanvalue data of small regions which are unnecessary regions (hereinafter,referred to as an unnecessary region cluster), by which the unnecessaryregion cluster is identified.

More specifically, first, a centroid ω_(a) bar (the bar indicates thatthe symbol “-” representing “mean” is described above ω) of each cluster“a” (a is the cluster number, and in the case of FIG. 7 a=1, 2, and 3)is computed by the following equation (1), using the sum total ω_(a) ofhue mean values in the cluster “a” and the number of pieces of databelonging to the cluster “a” (i.e., the number of small regions) m_(a).

$\begin{matrix}{{\overset{\_}{\omega}}_{a} = \frac{\omega_{a}}{m_{a\;}}} & (1)\end{matrix}$

In equation (1),

$\omega_{a} = {\sum\limits_{i = 1}^{n}{f\left( h_{i} \right)}}$

where

${f\left( h_{i} \right)} = \left\{ \begin{matrix}{h_{i}:{{a\_ min} \leq h_{i} \leq {a\_ max}}} \\{{0:{h_{i} < {a\_ min}}},{{a\_ max} < h_{i}}}\end{matrix} \right.$

In addition, the symbol i represents the label number of a small region.The symbol n represents the number of labels, i.e., the number of smallregions in the image. The symbol h_(i) represents the hue mean value ofthe small region i. The symbol a_min represents the minimum value of thehue mean values of the cluster “a”, and the symbol a max represents themaximum value of the hue mean values of the cluster “a”. Furthermore,

$m_{a} = {\sum\limits_{i = 1}^{n}I_{i}}$

where

$I_{i} = \left\{ \begin{matrix}{1:{{a\_ min} \leq h_{i} \leq {a\_ max}}} \\{{0:{h_{i} < {a\_ min}}},{{a\_ max} < h_{i}}}\end{matrix} \right.$

In the histogram of hue mean values illustrated in FIG. 7, the more itgoes to the left on the hue axis (horizontal axis), the more intense ayellow-based color. Hence, a residue determination threshold valueindicating a residue hue range is computed from training data which isacquired in advance, and a cluster whose centroid is located more on theleft side of the graph than the residue determination threshold value isdetermined to be an unnecessary region cluster. For example, in FIG. 7,cluster 1 is determined to be an unnecessary region cluster.

Furthermore, the mucosal region extracting unit 111 assigns anunnecessary region flag (2: unnecessary region) to small regionsbelonging to the unnecessary region cluster among the small regions thatare assigned the mucosal region flag (0: mucosal region) in thedetermination result list.

Note that in the determination of unnecessary regions, the same processmay be performed using the mean values of feature data other than hue.

At subsequent step S206, the region-of-interest setting unit 110acquires coordinate information of mucosal regions (regions other thanthe unnecessary regions) in the image, based on the labeled image andthe flag information in the determination result list, and outputs thecoordinate information of mucosal regions as a region of interest.Thereafter, processing returns to the main routine.

At step S30 subsequent to step S20, the linear convex region extractingunit 120 extracts, from the region of interest in the image, linearregions each having a predetermined number or more ofcontinuously-arranged pixels whose pixel values are higher than those oftheir neighboring pixels. FIG. 8 is a flowchart illustrating the detailsof processes performed by the linear convex region extracting unit 120.

At step S301, the convex shape high-frequency component computing unit121 computes the strengths of projection shape's high-frequencycomponents. More specifically, first, the convex shape high-frequencycomponent computing unit 121 creates a G-component image using Gcomponents included in the pixel values of the respective pixels in theintraluminal image. The reason that the G components are used is that,as described above, the G components are close to the blood absorptionband and thus provides the best representation of the structures ofobjects. Note that although in the first embodiment the G components areused, instead, for example, other color components, or a luminancevalue, a color difference (YCbCr conversion), hue, saturation, intensity(HSI conversion), or a color ratio, which is obtained by converting apixel value (R, G, and B components) may be used.

Subsequently, the convex shape high-frequency component computing unit121 performs a top-hat transform by grayscale morphology (reference: ByHidefumi Kobatake, “Morphology”, CORONA PUBLISHING CO., LTD., pp.103-104) on the G-component image. Here, the top-hat transform refers toa process in which, as illustrated in FIG. 9, an output y=f_(g)(x) whichis obtained by performing a low-pass filter process on an originalsignal output y=f(x) is subtracted from the original signal output f(x),and the output of a high-pass filter process y=f(x)−f_(g)(x) is given.In addition, for the low-pass filter process performed at this time, amorphological opening process is performed. The opening processcorresponds to a process of tracing the provided signal f(x) from thebottom by spherical structural elements such as those illustrated inFIG. 9. At this time, in a portion where the signal changes slowly, thesignal output y=f(x) is traced accurately, but in a portion with apositive pulse, a sphere (structural element) cannot fit completely andthus the signal output goes out of a traceable range. Hence, by theopening process, the output y=f_(g)(x) where the positive pulses areremoved can be obtained. By subtracting the output y=f_(g)(x) from theoriginal signal output y=f(x), as a result, high-frequency components inthe convex-shaped region which are positive pulse outputs can beacquired.

Note that the strengths of projection shape's high-frequency componentsmay be computed using a high-pass filter process in a Fourier space,Difference of Gaussian (DoG), etc., in addition to the above.

At subsequent step S302, the isolated-point excluding unit 122 extractsisolated points based on the strengths of projection shape'shigh-frequency components computed at step S301, and excludes theisolated points as normal villi.

Specifically, the isolated-point excluding unit 122 first performs athreshold value process on the strengths of projection shape'shigh-frequency components, and thereby creates a binarized image. Atthis time, the isolated-point excluding unit 122 provides 0 to a pixelwhose strength of a projection shape's high-frequency component is lessthan or equal to a predetermined threshold value, and provides 1 to apixel whose strength is greater than or equal to the threshold value.Note that for the threshold value, for example, the value of apredetermined ratio to one representative value (maximum value or meanvalue) which is computed from the strengths of projection shape'shigh-frequency components in the region of interest is set.Alternatively, as the threshold value, a fixed value greater than 0 maybe set. In the latter case, the processes of calculating arepresentative value for projection shape's high-frequency componentsand calculating a threshold value from the representative value can beomitted. Note that the binarized image created by this process serves asan image from which convex-shaped high-frequency regions have beenextracted.

Subsequently, the isolated-point excluding unit 122 performs particleanalysis and thereby excludes isolated points from the binarized image.In the first embodiment, as the particle analysis, analysis is performedbased on the areas of target regions (particles) in the binarized image.Specifically, the isolated-point excluding unit 122 labels regions withthe pixel value 1 in the binarized image and thereby creates a labeledimage that is assigned label numbers 1 to s (s is the number of labels).Then, by scanning the labeled image, the number of pixels is counted foreach label number, by which an area list is created. Here, the area listis composed of the label numbers and areas for the respective labelnumbers, and the size of the area list corresponds to the number oflabels s. The isolated-point excluding unit 122 further performs athreshold value process on the areas of labeled regions associated withthe respective label numbers, using a predetermined threshold value, anddetermines a labeled region with a label number whose corresponding areais less than or equal to the threshold value, to be an isolated point.The label number of the labeled region determined to be an isolatedpoint is set to 0.

Note that, when an isolated-point exclusion process is performed,particle analysis may be performed using the perimeter length of atarget region, a Feret diameter (maximum Feret diameter), an absolutemaximum length indicating the maximum value of a distance between anytwo points on a contour of a target region, an equivalent circulardiameter indicating the diameter of a circle having an equal area to atarget region, or the like, in addition to the area of a target region.

At step S303, the thinning unit 123 thins convex-shaped high-frequencyregions. More specifically, the thinning unit 123 performs a thinningprocess on regions whose label numbers are 1 or more in the labeledimage where isolated points have been excluded by the isolated-pointexcluding unit 122. To do so, the thinning unit 123 first provides thepixel value 0 to a region with the label number 0 and provides the pixelvalue 1 to a region with the label number 1 or more in the labeledimage, and thereby creates a binarized image. Subsequently, the thinningunit 123 performs a thinning filtering process on the binarized image.The thinning filtering process refers to a process in which patterns M1to M8 of local regions including 3×3 pixels such as those illustrated inFIG. 10 are sequentially searched for in the binarized image, andcentral pixels are deleted (reference: University of Tokyo Press,“Handbook of Image Analysis”, pp. 577-578 (thinning and shrinking)).Specifically, first, the pattern M1 is searched for in the binarizedimage, and the pixel value of the central pixel is changed to 0. Notethat “*” illustrated in FIG. 10 represents a pixel that does not need tobe considered. Then, the pattern M2 is searched for in the binarizedimage, and the pixel value of the central pixel is change to 0. Such aprocess is repeated for up to the pattern M8, which serves as one cycle.This cycle is repeated until there is no more point to delete.

Furthermore, at step S304, the linear convex region extracting unit 120outputs the binarized image having been subjected to the thinningprocess (hereinafter, also referred to as a linear image). Thereafter,processing returns to the main routine.

At step S40 subsequent to step S30, the intra-region curvature featuredata computing unit 130 computes curvature feature data based on thecurvatures of arcs along the linear regions. FIG. 11 is a flowchartillustrating the details of processes performed by the intra-regioncurvature feature data computing unit 130.

At step S401, the size feature data computing unit 131 computes, as arcsize feature data, the curvatures of an arc along a linear region whichis observed in the linear image, and distance information correspondingto an imaging distance from an imaging position (capsule endoscope) tothe arc. The process performed by the size feature data computing unit131 will be described in detail with reference to FIG. 12.

At step S4011, the size feature data computing unit 131 labels acontinuous linear region in the linear image. In the first embodiment,labeling for an eight-connection connecting component is performed.Specifically, the size feature data computing unit 131 first performs araster scan on the linear image and searches for a pixel that is notassigned a label number among the pixels with the pixel value 1. Then,when a pixel that is not assigned a label number is found, the pixel isset as a pixel of interest.

When a pixel adjacent above or present at the upper left of the pixel ofinterest has a label number, the size feature data computing unit 131assigns the label number of the pixel adjacent above or present at theupper left to the pixel of interest. Thereafter, when the label numberof a pixel adjacent to the left side of the pixel of interest differsfrom the label number of the pixel of interest, the fact that the labelnumbers of the pixel of interest and the pixel adjacent to the left sidethereof belong to the same connecting component is recorded in a look-uptable which is prepared in advance.

In addition, when the label numbers of the pixels adjacent above andpresent at the upper left of the pixel of interest are 0 (no labelnumber) and the pixel adjacent to the left side has a label number, thelabel number of the pixel adjacent to the left side is assigned to thepixel of interest.

Furthermore, when a label number is not assigned to any of the pixelsadjacent above, adjacent to the left side, and present at the upper leftof the pixel of interest, the size feature data computing unit 131assigns a new label number to the pixel of interest.

The size feature data computing unit 131 performs these processes on allpixels in the linear image. Then, finally, a raster scan is performedagain, and by referring to the look-up table, the smallest label numberamong label numbers that are assigned to a pixel group belonging to thesame connecting component is selected and reassigned to the pixel group(reference: Computer Graphic Arts Society, “Digital Image Processing”,pp. 181-182).

At step S4012, as illustrated in FIG. 13, the curvature computing unit131 a computes the curvatures of one or more arcs from each section of alinear region L that is delimited by endpoints P_(t) of the linearregion L and/or intersection points P_(c) of linear regions L in alinear image G1 (i.e., between endpoints, between intersection points,or between an endpoint and an intersection point).

More specifically, the curvature computing unit 131 a scans the linearimage and thereby detects a pixel at an endpoint P_(t) of a linearregion or an intersection point P_(c) of linear regions. Then, asillustrated in FIG. 14, the detected pixel at the endpoint P_(t) orintersection point P_(c) is set as a start point pixel M(x_(s), y_(s)).Furthermore, a pixel that is reached by following the linear region L(i.e., pixels assigned the same label number) by a predetermined numberof pixels Δs from the start point pixel M(x_(s), y_(s)) is set as an endpoint pixel N(x_(e), y_(e)).

Subsequently, the curvature computing unit 131 a computes a slope of atangent line at the start point pixel M and a slope of a tangent line atthe end point pixel N. In the present embodiment, the slopes of therespective tangent lines are represented by angles DEG_(s) and DEG_(e)formed with an x-axis. The angle DEG_(i) (i=s and e) formed between eachtangent line and the x-axis is computed using the following equation (2)by setting two pixels (coordinates (x_(e), y_(e)) and (x_(b), y_(b)))which are separated forward and backward from the start point pixel M(or the end point pixel N) along the linear region L by a predeterminednumber of pixels.

$\begin{matrix}{{DEG}_{i} = {{- \left\{ {\tan^{- 1}\left( \frac{y_{b} - y_{a}}{x_{b} - x_{a}} \right)} \right\}} \times 180 \times \frac{1}{\pi}}} & (2)\end{matrix}$

Note that when the upper left of the linear image is set to thecoordinates (0, 0) and each row is scanned in the right direction fromthe upper left, the coordinates (x_(a), y_(a)) are forward coordinatesof the start point pixel M (or the end point pixel N), and thecoordinates (x_(b), y_(b)) are backward coordinates of the start pointpixel M (or the end point pixel N). Note, however, that when coordinatesgo out of the linear region L by the separation from the start pointpixel M (or the end point pixel N) by the predetermined number of pixels(when coordinates go beyond the endpoint P_(t) or the intersection pointP_(a)), the coordinates of the endpoint P_(t) or the intersection pointP_(c) of the linear region L are set to either one of theabove-described coordinates (x_(a), y_(a)) and (x_(b), y_(b)).

Subsequently, the curvature computing unit 131 a computes a centralangle α of an arc between the start point pixel M and the end pointpixel N. The central angle α corresponds to the difference between theangle DEG, and the angle DEG_(e), i.e., DEG_(e)−DEG_(s).

The curvature computing unit 131 a further computes a curvature κ of thearc between the start point pixel M and the end point pixel N. Here, thecurvature κ is the reciprocal of a curvature radius R of the arc, andthe number of pixels Δs corresponding to the length of the arc can beapproximated by R×α×(π/180), and thus, the curvature κ is given by thefollowing equation (3):

$\begin{matrix}{\kappa = {\frac{1}{R} = \frac{\alpha \times \frac{\pi}{180}}{\Delta\; s}}} & (3)\end{matrix}$

As illustrated in FIG. 15, the curvature computing unit 131 a repeats acomputation process for the curvature κ (θ₁, κ₂, . . . ) by shifting thestart point pixel M (M₁, M₂, . . . ) and the end point pixel N (N₁, N₂,. . . ) on the linear region L. Then, when the end point pixel N goesbeyond the endpoint P_(t) of the linear region L or the intersectionpoint P_(c) of the linear regions L, the curvature computing unit 131 aends the computation process for the curvature κ, and assigns a labelnumber which is incremented from 1 in turn, to the linear region L whosecurvature κ has been computed.

The curvature computing unit 131 a performs such computation of thecurvature κ and an assignment of a label number on all linear regions Lthat are detected from the linear image G1 illustrated in FIG. 13. Thenumbers in parentheses illustrated in FIG. 13 indicate label numbersassigned to the respective linear regions L.

At subsequent step S4013, the curvature representative value computingunit 131 b computes a representative value of the curvature of an arcfor each linear region that is assigned the same label number by thecurvature computing unit 131 a. Note that although in the firstembodiment a median is computed as the representative value, the meanvalue, maximum value, minimum value, etc., of the curvature may becomputed as the representative value.

At subsequent step S4014, the distance information computing unit 131 ccomputes distance information in a depth direction from an imagingposition to the linear regions. Specifically, the distance informationcomputing unit 131 c acquires R component values that have lowhemoglobin absorption among the pixel values of the respective pixelsand thus provide the best representation of the shape of a mucosalsurface layer, from a portion of the original intraluminal imagecorresponding to the region of interest, and creates an R-componentimage.

Note that at this step S4014, distance information may be acquired froman image created by other techniques, provided that the image has only asmall influence on the microstructures of villi on the mucosa. Forexample, an image having been subjected to an opening process bygrayscale morphology which is described in the process of the convexshape high-frequency component computing unit 121 may be used.

Subsequently, the distance information computing unit 131 c computesdistance information for the position of each arc whose curvatures arecomputed by the curvature representative value computing unit 131 b.Specifically, R component values for the positions of pixels on an arcare acquired, and a representative value such as the mean value, median,minimum value, or maximum value of the R component values is acquiredfor each arc. In the first embodiment, the representative value of the Rcomponent values is used as distance information corresponding to animaging distance. Note that in this case the larger the value of thedistance information the shorter the imaging distance, and the smallerthe value of the distance information the longer the imaging distance.Note also that an R component value for the position of a pixel at thecenter of curvature of an arc or R component values of an inner regionof a circular region or fan-shaped region which is made up of the centerof curvature of an arc and a curvature radius may be used as distanceinformation of the arc.

Furthermore, at step S4015, the size feature data computing unit 131outputs the curvatures and distance information of the arcs as arc sizefeature data. Thereafter, processing returns to the main routine.

At step S402 subsequent to step S401, the frequency distributioncreating unit 132 creates a frequency distribution of the feature dataoutputted from the size feature data computing unit 131. Morespecifically, a frequency distribution consisting of two axes, arccurvature and distance information, is created. At this time, thefrequency distribution creating unit 132 also computes an amount ofstatistics such as variance, from the created frequency distribution.

Furthermore, at step S403, the intra-region curvature feature datacomputing unit 130 outputs, as curvature feature data, the frequencydistribution consisting of two axes, arc curvature and distanceinformation. Thereafter, processing returns to the main routine.

At step S50 subsequent to step S40, the abnormality determining unit 140determines whether there is an abnormal portion in the region ofinterest, based on the curvature feature data.

Here, arc-shaped regions seen in the intraluminal image include, forexample, an abnormal portion where villi on the mucosal surface areswollen, a bubble region where intraluminal fluid forms a bubble shape,and a contour of a structure such as a mucosal groove (hereinafter,referred to as a mucosal contour). In the following, the swollen villiare referred to as abnormal villi.

FIG. 16A is a schematic diagram illustrating a normal villus model. Anormal villus is normally circular in planar shape viewed in a directionof the normal to the mucosal surface, and forms as a whole a shape wherea central portion protrudes. Thus, in the intraluminal image, thecentral portion of the villus and its neighboring portion have thehighest luminance. Hence, when convex regions having higher luminancevalues than their neighboring pixels are extracted from an intraluminalimage where normal villi are displayed, as illustrated in FIG. 16B,dot-like convex regions PR1 corresponding to the central portions of thevilli are obtained. Such dot-like convex regions PR1 are excluded inadvance as isolated points from the linear image.

FIG. 17A is a schematic diagram illustrating an abnormal villus model. Aplanar shape of an abnormal villus viewed in a direction of the normalto the mucosal surface forms a flat shape like an ellipse. Thus, in theintraluminal image, a contour portion according to the orientation ofthe villus with respect to an imaging direction (a contour portion witha high curvature or a contour portion with a low curvature) has a highluminance. Therefore, when convex regions are extracted from anintraluminal image where abnormal villi are displayed, as illustrated inFIG. 17B, linear regions PR2 that form arc shapes having curvaturesaccording to the imaging direction are obtained. Hence, a distributionof the curvatures of arcs along the linear regions PR2 falls within asomewhat narrow range, as illustrated in FIG. 17C. In addition, when aminimum value of the curvature is adopted as a representative value ofthe curvature of an arc along each linear region PR2, the curvatures aredistributed leaning toward the small value side (see a solid line). Onthe other hand, when a maximum value of the curvature is adopted as arepresentative value of the curvature, the curvatures are distributedleaning toward the large value side (see a dashed line).

FIG. 18A is a schematic diagram illustrating a bubble model. In a bubbleregion, each individual bubble forms a substantially spherical shape. Inaddition, since the entire bubble surface is a high reflective region,in the intraluminal image a bubble contour portion has a high luminance.Therefore, when convex regions are extracted from an intraluminal imagewhere a bubble region is displayed, as illustrated in FIG. 18B, linearregions PR3 of various sizes that form a substantially circular shapeare obtained. Hence, a distribution of the curvatures of arcs along thelinear regions PR3 greatly varies, as illustrated in FIG. 18C.

In addition, the curvatures of arcs along mucosal contours generallytake small values compared to abnormal villi and bubble regions.

FIG. 19 is a schematic diagram illustrating a frequency distributionconsisting of two axes, arc curvature and distance information. Thecurvatures of arcs along linear regions representing mucosal contoursare mostly independent of the imaging distance, and have small valuesand are distributed in a narrow range. In addition, the curvatures ofarcs along linear regions representing a bubble region are independentof the imaging distance and are distributed in a wide range. On theother hand, a distribution range of the curvatures of arcs along linearregions representing abnormal villi changes according to the imagingdistance. Specifically, when the imaging distance increases (the valueof imaging information decreases), the curvatures are distributed on thelarge value side (on the steep arc curve side), and when the imagingdistance decreases (the value of imaging information increases), thecurvatures are distributed on the small value side (on the gentle arccurve side).

Hence, the abnormality determining unit 140 makes a determination asfollows.

First, when the variance of curvature for given distance informationL_(arb) is greater than a predetermined threshold value (variancethreshold value) TH₁ which is determined in advance according to thedistance information L_(arb), a bubble region is displayed in the regionof interest and thus the abnormality determining unit 140 determinesthat there is no abnormal portion.

In addition, in the case in which the variance of curvature for thedistance information L_(arb) is less than or equal to the variancethreshold value TH₁, when the curvatures are distributed in a rangesmaller than a predetermined threshold value (curvature threshold value)TH₂, mucosal contours are displayed in the region of interest and thusthe abnormality determining unit 140 determines that there is noabnormal portion.

On the other hand, in the case in which the variance of curvature forthe distance information L_(arb) is less than or equal to the variancethreshold value TH₁, when the curvatures are distributed in a rangegreater than or equal to the curvature threshold value TH₂, abnormalvilli are displayed in the region of interest and thus the abnormalitydetermining unit 140 determines that there is an abnormal portion.

At subsequent step S60, the calculating unit 100 outputs a result of thedetermination as to whether there is an abnormal portion which is madeat step S50. Accordingly, the control unit 10 displays the result of thedetermination on the display unit 40 and records the result of thedetermination in the recording unit 50 so as to be associated with theimage data of the processing target intraluminal image.

As described above, according to the first embodiment, a mucosal regionwhich is extracted from an intraluminal image is set as a region ofinterest, and high-frequency components included in convex regions, eachof which is composed of a pixel group having higher pixel values thanits neighboring pixels, are extracted from the region of interest. Then,as curvature feature data, curvatures and distance information arecomputed for linear regions obtained by thinning the high-frequencycomponents, and an object displayed in the region of interest isidentified based on a frequency distribution of the feature data.Accordingly, erroneous detection (overdetection) of mucosal contours,bubbles, etc., is suppressed, and also changes in the size ofmicrostructures of villi according to the imaging distance and changesin shape according to the orientation of villi with respect to theimaging direction are dealt with, by which abnormal villi are accuratelyidentified, enabling to determine whether there is an abnormal portion.

Modification 1-1

Although, in the above-described first embodiment, the entire remainingregion obtained after removing unnecessary regions from an intraluminalimage is set as one region of interest, each of a plurality of regionsinto which the remaining region is divided may be set as a region ofinterest. In this case, it becomes possible to identify at whichlocation in the intraluminal image an abnormal portion is present. Adivision method is not particularly limited, and for example, aremaining region obtained after removing unnecessary regions may besimply divided into a matrix pattern. Alternatively, distanceinformation corresponding to an imaging distance may be acquired fromthe pixel value of each pixel in a remaining region obtained afterremoving unnecessary regions, and the region may be divided on ahierarchy-by-hierarchy basis, the distance information being dividedinto hierarchies. In this case, the process at step S4014 illustrated inFIG. 12 can be omitted.

Modification 1-2

Next, a modification 1-2 of the first embodiment will be described.

FIG. 20 is a block diagram illustrating an image processing apparatusaccording to the modification 1-2. As illustrated in FIG. 20, an imageprocessing apparatus 1-2 of the modification 1-2 includes a calculatingunit 100-2 instead of the calculating unit 100 illustrated in FIG. 1.

The calculating unit 100-2 includes a linear convex region extractingunit 150 instead of the linear convex region extracting unit 120illustrated in FIG. 1. The linear convex region extracting unit 150includes a ridge-shape extracting unit 151 that extracts, as aridge-shaped region, a pixel group whose pixel values change in aridge-shaped manner; an isolated-point excluding unit 152 that excludes,as an isolated point, a pixel group that is isolated from itsneighboring pixels, from the extracted ridge-shaped region; and athinning unit 153 that thins the ridge-shaped region obtained afterexcluding the isolate points, and extracts a linear convex region.

The overall operation of the calculating unit 100-2 is similar to thatillustrated in FIG. 2, but only the content of processes performed bythe linear convex region extracting unit 150 at step S30 is different.FIG. 21 is a flowchart illustrating the details of processes performedby the linear convex region extracting unit 150.

First, at step S311, the ridge-shape extracting unit 151 extracts aridge-shaped region from a region of interest. More specifically, theridge-shape extracting unit 151 sequentially sets pixels of interest inthe region of interest, and calculates a maximum direction (hereinafter,referred to as a maximum gradient direction) and a minimum direction(hereinafter, referred to as a minimum gradient direction) of a gradientof a pixel value for each pixel of interest. Then, curvatures of a shapewhose pixel values change are computed for each of the maximum gradientdirection and the minimum gradient direction, and the ridge-shapedregion is detected based on a ratio between these curvatures.

To do so, the ridge-shape extracting unit 151 first solves an eigenvalueequation of a Hessian matrix H shown in the following equation (4) for apixel of interest, and thereby computes, as eigenvalues, a curvature k₁for the maximum gradient direction and a curvature k₂ for the minimumgradient direction.det(H−λE)=0  (4)

In equation (4), the Hessian matrix H is given by the following equation(5):

$\begin{matrix}{H = \begin{bmatrix}\frac{\partial^{2}{f\left( {x,y} \right)}}{\partial x^{2}} & \frac{\partial^{2}{f\left( {x,y} \right)}}{{\partial x}{\partial y}} \\\frac{\partial^{2}{f\left( {x,y} \right)}}{{\partial y}{\partial x}} & \frac{\partial^{2}{f\left( {x,y} \right)}}{\partial y^{2}}\end{bmatrix}} & (5)\end{matrix}$

In addition, the symbol E is the identify matrix and λE is as shown inthe following equation (6):

$\begin{matrix}{{\lambda\; E} = \begin{bmatrix}k_{1} & 0 \\0 & k_{2}\end{bmatrix}} & (6)\end{matrix}$

Subsequently, a Gaussian curvature κ_(G) and a mean curvature A_(k)which are given by the following equations (7-1) and (7-2) are computedfrom the curvatures k₁ and k₂.K _(G) =k ₁ ×k ₂  (7-1)A _(k)=(k ₁ +k ₂)/2  (7-2)

At this time, a pixel group where A_(k)<0 and K_(G)≈0, i.e., a pixelgroup where the absolute value |K_(G)| is less than or equal to apredetermined threshold value, is determined to be a ridge-shapedregion. The ridge-shape extracting unit 151 assigns the pixel value 1 tothe pixel group determined to be a ridge-shaped region, and assigns thepixel value 0 to other pixels, and thereby creates a binarized image.

Note that after computing the curvatures k₁ and k₂, a ridge-shapedregion may be determined using any publicly known technique thatrecognizes arbitrary three-dimensional surface shapes like, for example,shape index and curvedness (reference: Chitra Dorai, Anil K. Jain,“COSMOS—A Representation Scheme for Free-Form Surfaces”).

At subsequent step S312, the isolated-point excluding unit 152 extractsisolated points from the ridge-shaped region which is extracted at stepS311, and excludes the isolated points as normal villi. Morespecifically, a pixel group where the number of continuously-arrangedpixels is less than or equal to a predetermined value (i.e., a region ofa size less than or equal to a predetermined value) is extracted as anisolated point from the binarized image created at step S311. Then, thepixel value 0 is assigned to the pixel group extracted as an isolatedpoint.

At subsequent step S313, the thinning unit 153 thins the ridge-shapedregion obtained after excluding the isolated points. More specifically,the thinning unit 153 performs a thinning filtering process using thepatterns M1 to M8 illustrated in FIG. 10, on the binarized image createdat step S312. Note that the detailed content of the thinning filteringprocess is the same as that of the first embodiment.

Furthermore, at step S314, the linear convex region extracting unit 150outputs the binarized image having been subjected to the thinningprocess. Thereafter, processing returns to the main routine.

Modification 1-3

In the first embodiment, for the process of thinning a convex-shapedhigh-frequency region which is extracted from a region of interest (seestep S303 in FIG. 8), various publicly known techniques can be appliedin addition to the above-described method. A Hilditch thinning algorithmwhich is one of the publicly known techniques will be described below.

First, as a first step, in a binarized image from which a convex-shapedhigh-frequency region has been extracted, pixels that satisfy sixconditions shown below are deleted sequentially as boundary pixels,among pixels P_(k) which are target pixels for thinning. Here, k is thepixel number of a pixel in the binarized image (k is a natural number).In addition, the pixel value of the pixel P_(k) is represented byB(P_(k)) (B(P_(k))=1). In this case, the first step corresponds to aprocess of replacing the pixel value B(P_(k)) from 1 to −1. Note thatthe pixel value of a non-target pixel for thinning is B(P_(k))=0.

Condition 1: A pixel of interest is a target pixel for thinning. Thatis, the following equation (a1) is satisfied.B(P _(k))=1  (a1)

Condition 2: A pixel value of any one of pixels that are adjacent to thepixel of interest in vertical and horizontal directions is 0. That is,the following equation (a2) is satisfied.

$\begin{matrix}{{\sum\limits_{k = 1}^{4}\left( {1 - {{B\left( P_{{2k} - 1} \right)}}} \right)} \geq 1} & \left( {a\; 2} \right)\end{matrix}$

Condition 3: Not an end point. That is, the following equation (a3) issatisfied.

$\begin{matrix}{{\sum\limits_{k = 1}^{8}{{B\left( P_{k} \right)}}} \geq 2} & ({a3})\end{matrix}$

Condition 4: Not an isolated point. That is, the following equation (a4)is satisfied. Here, a result obtained by a sequential process isrepresented by B(P_(k)) and a result obtained when an immediatelyprevious raster operation is completed is represented by B′(P_(k)), bywhich the results are distinguished from each other. Note that the pixelvalue B′(P_(k)) does not take −1.

$\begin{matrix}{{\sum\limits_{k = 1}^{8}{B^{\prime}\left( P_{k} \right)}} \geq 1} & ({a4})\end{matrix}$

Condition 5: Connectivity is maintained. That is, any of patterns M11 toM14 illustrated in FIG. 22 is applied. Note that in FIG. 22 the pixelrepresented by “*” takes the pixel value 1 or −1. This condition isrepresented by a conditional expression shown in the following equation(a5):

$\begin{matrix}{{\sum\limits_{k = 1}^{4}\left\{ {1 - {{B\left( P_{{2k} - 1} \right)}} - {\left( {1 - {{B\left( P_{{2k} - 1} \right)}}} \right)\left( {1 - {{P\left( P_{2k} \right)}}} \right)\left( {1 - {{P\left( P_{{2k} + 1} \right)}}} \right)}} \right\}} = 1} & ({a5})\end{matrix}$

Condition 6: For a line segment with a line width of 2, only one of themis deleted. That is, the result B′(P_(k)) obtained when an immediatelyprevious raster operation is completed applies to any one of patternsM21 to M24 illustrated in FIG. 23. This condition is represented by aconditional expression shown in the following equation (a6):

$\begin{matrix}{{\sum\limits_{k = 1}^{4}\left\{ {1 - {B^{\prime}\left( P_{{2k} - 1} \right)} - {\left( {1 - {B^{\prime}\left( P_{{2k} - 1} \right)}} \right)\left( {1 - {B^{\prime}\left( P_{2k} \right)}} \right)\left( {1 - {B^{\prime}\left( P_{{2k} + 1} \right)}} \right)}} \right\}} = 1} & ({a6})\end{matrix}$

Next, as a second step, the pixel values of the pixels having beendeleted sequentially as boundary pixels (i.e., the pixels whose pixelvalues B(P_(k)) have been replaced from 1 to −1) are replaced by thepixel value of a non-target region B(P_(k))=0.

These first and second steps are repeated until there is no morereplacement to the pixel value of a non-target region. By that, thinningof a target region is performed.

Second Embodiment

Next, a second embodiment of the present invention will be described.

FIG. 24 is a block diagram illustrating a configuration of an imageprocessing apparatus according to the second embodiment. As illustratedin FIG. 24, an image processing apparatus 2 according to the secondembodiment includes a calculating unit 200 instead of the calculatingunit 100 illustrated in FIG. 1. The calculating unit 200 includes aregion-of-interest setting unit 110, a linear convex region extractingunit 120, an intra-region curvature feature data computing unit 210, andan abnormality determining unit 220. Note that the configurations andoperation of the region-of-interest setting unit 110 and the linearconvex region extracting unit 120 are the same as those of the firstembodiment. Note also that the configurations and operation of thoseunits of the image processing apparatus other than the calculating unit200 are also the same as those of the first embodiment.

The intra-region curvature feature data computing unit 210 includes ashape feature data computing unit 211 that computes the curvatures ofone or more arcs along each linear region which is extracted by thelinear convex region extracting unit 120, and computes variation inthose curvatures as feature data; and a frequency distribution creatingunit 212 that creates a frequency distribution of the feature data. Morespecifically, the shape feature data computing unit 211 includes acurvature computing unit 211 a that computes the curvatures of one ormore arcs from each section of a linear region that is delimited byendpoints of the linear region and/or intersection points of linearregions; and a curvature standard deviation computing unit 211 b thatcomputes a standard deviation of the curvatures of one or more arcswhich are computed from each section.

The abnormality determining unit 220 determines whether there is anabnormal portion, based on the variation in the curvatures.

Next, the operation of the image processing apparatus 2 will bedescribed.

The overall operation of the image processing apparatus 2 is similar tothat illustrated in FIG. 2, but the contents of processes performed bythe intra-region curvature feature data computing unit 210 at step S40and processes performed by the abnormality determining unit 220 at stepS50 are different from those of the first embodiment.

FIG. 25 is a flowchart illustrating processes performed by theintra-region curvature feature data computing unit 210. First, at stepS421, the shape feature data computing unit 211 computes, as featuredata representing a shape of a linear region which is extracted at stepS30, a variation in the curvatures of an arc along the linear region.The processes performed by the shape feature data computing unit 211will be described in detail with reference to FIG. 26. Note that stepsS4211 and S4212 illustrated in FIG. 26 correspond to steps S4011 andS4012 illustrated in FIG. 12, respectively. The process at step S4212 isperformed by the curvature computing unit 211 a instead of the curvaturecomputing unit 131 a (see FIG. 1).

At step S4213 subsequent to step S4212, the curvature standard deviationcomputing unit 211 b computes, for each section of the linear region, astandard deviation of the curvatures of one or more arcs. Specifically,the curvature standard deviation computing unit 211 b computes astandard deviation σ which is given by the following equation (8), fromm curvatures κ_(i) (i=1 to m) which are computed for an arc having thesame label number assigned by the curvature computing unit 211 a.

$\begin{matrix}{\sigma = \sqrt{\frac{1}{m}{\sum\limits_{i = 1}^{m}\left( {\kappa_{i} - \overset{\_}{\kappa}} \right)^{2}}}} & (8)\end{matrix}$

In equation (8), the κ bar where a bar is provided above κ is the meanvalue of the curvature κ_(i), and is given by the following equation(9):

$\begin{matrix}{\overset{\_}{\kappa} = {\frac{1}{m}{\sum\limits_{i = 1}^{m}\kappa_{i}}}} & (9)\end{matrix}$

At subsequent step S4214, the shape feature data computing unit 211outputs the standard deviation computed for each section of the linearregion, as feature data representing the shape of the linear region.Thereafter, processing returns to the main routine.

At step S422 subsequent to step S421, the frequency distributioncreating unit 212 creates, as a frequency distribution of the featuredata, a frequency distribution of the standard deviations outputted fromthe shape feature data computing unit 211.

Furthermore, at step S423, the intra-region curvature feature datacomputing unit 210 outputs the frequency distribution of the standarddeviations as curvature feature data. Thereafter, processing returns tothe main routine.

Next, processes performed by the abnormality determining unit 220 willbe described.

The abnormality determining unit 220 determines whether there is anabnormal portion in a region of interest, based on the frequencydistribution of standard deviations which is curvature feature data.

Here, when there are abnormal villi in the region of interest, asillustrated in FIG. 27A, flat linear regions PR4 appear in a linearimage created from the region of interest. As illustrated in FIG. 27B,the curvatures of an arc along such a linear region PR4 take variousvalues according to the position in the linear region PR4. Hence,variation in the curvatures is relatively large. Note that a vectorV_(i) (i=1, 2, . . . ) illustrated in FIG. 27B is a vector representingthe magnitude of the curvature of the arc. Therefore, as illustrated inFIG. 27C, the standard deviations of curvatures computed for the linearregions PR4 are distributed in a relatively large-value range.

On the other hand, when there is a bubble region in the region ofinterest, as illustrated in FIG. 28A, circular linear regions PR5 appearin a linear image created from the region of interest. As illustrated inFIG. 28B, in such a linear region PR5, the curvatures of arcs have asubstantially uniform value and thus variation in the curvatures issmall. Therefore, as illustrated in FIG. 28C, the standard deviations ofcurvatures computed for the linear regions PR5 are distributed in asmall-value range (0 or near 0).

Hence, the abnormality determining unit 220 determines whether there isan abnormal portion, based on the frequency distribution of standarddeviations. Specifically, when the frequency distribution of standarddeviations leans toward a range greater than a predetermined thresholdvalue, it is determined that there is an abnormal portion in the regionof interest.

As described above, according to the second embodiment, a mucosal regionwhich is extracted from an intraluminal image is set as a region ofinterest, and high-frequency components included in convex regions, eachof which is composed of a pixel group having higher pixel values thanits neighboring pixels, are extracted from the region of interest. Then,for linear regions obtained by thinning the high-frequency components, afrequency distribution of standard deviations of curvatures of arcsalong the linear regions is computed, and it is determined whether thereis an abnormal portion in the region of interest, based on the frequencydistribution of standard deviations. Accordingly, abnormal villi andbubble regions are accurately identified, enabling to improve theaccuracy of determination of an abnormal portion.

In addition, according to the second embodiment, since a determinationis made without using an imaging distance (distance information) to anobject in the region of interest, a calculation process can besimplified.

Third Embodiment

Next, a third embodiment of the present invention will be described.

FIG. 29 is a block diagram illustrating a configuration of an imageprocessing apparatus according to the third embodiment. As illustratedin FIG. 29, an image processing apparatus 3 according to the thirdembodiment includes a calculating unit 300 instead of the calculatingunit 100 illustrated in FIG. 1. The calculating unit 300 includes aregion-of-interest setting unit 110, a linear convex region extractingunit 120, an intra-region curvature feature data computing unit 310, andan abnormality determining unit 320. Note that the configurations andoperation of the region-of-interest setting unit 110 and the linearconvex region extracting unit 120 are the same as those of the firstembodiment. Note also that the configurations and operation of thoseunits of the image processing apparatus 3 other than the calculatingunit 300 are also the same as those of the first embodiment.

The intra-region curvature feature data computing unit 310 includes adirection feature data computing unit 311 that computes curvaturefeature data representing the central directions of one or more arcsalong each linear region which is extracted by the linear convex regionextracting unit 120; and a frequency distribution creating unit 312 thatcreates a frequency distribution of the curvature feature data. Of them,the direction feature data computing unit 311 includes a curvaturecentral direction computing unit 311 a that computes a direction goingtoward a curvature center from each arc (hereinafter, referred to as acurvature central direction); and a gradient direction computing unit311 b that computes a gradient direction of an object for the positionof each arc. Here, the gradient direction of an object indicates adirection in which the object (specifically, a mucosal structure) isinclined in a depth direction of an image.

The abnormality determining unit 320 determines whether there is anabnormal portion, based on the gradient direction of the mucosalstructure and the distribution of the curvature central directions.

Next, the operation of the image processing apparatus 3 will bedescribed.

The overall operation of the image processing apparatus 3 is similar tothat illustrated in FIG. 2, but the contents of processes performed bythe intra-region curvature feature data computing unit 310 at step S40and processes performed by the abnormality determining unit 320 at stepS50 are different from those of the first embodiment.

FIG. 30 is a flowchart illustrating processes performed by theintra-region curvature feature data computing unit 310. First, at stepS431, the direction feature data computing unit 311 computes, as featuredata representing a central direction of an arc, a curvature centraldirection of the arc and a gradient direction. The processes performedby the direction feature data computing unit 311 will be described indetail with reference to FIG. 31.

At step S4311, the curvature central direction computing unit 311 acomputes a curvature central direction of an arc along a linear region.Specifically, first, in the same manner as step S4012 illustrated inFIG. 12, the curvatures of each arc are computed, and furthermore,curvature radii R which are the reciprocal of the curvatures arecomputed.

Subsequently, the curvature central direction computing unit 311 aperforms the Hough transform using position coordinates (x, y) on thearc and the curvature radii R to create an approximate circle(reference: Computer Graphic Arts Society, “Digital Image Processing”,pp. 213-214). Here, the approximate circle obtained by the Houghtransform can be created by voting a circle with a radius R passingthrough the position coordinates (x, y) on the arc into a parameterspace consisting of the central coordinate (a, b) of the circle and theradius R, and by checking results of the voting.

Furthermore, the curvature central direction computing unit 311 acomputes a direction vector going toward the center of the approximatecircle from an arbitrary point (e.g., a central point) on the arc, andoutputs the direction vector as a curvature central direction.

At subsequent step S4312, the gradient direction computing unit 311 bcomputes a gradient direction of an object at the position of the arcalong the linear region. Specifically, the gradient direction computingunit 311 b acquires, from a portion of an original intraluminal imagecorresponding to a region of interest as a processing target, Rcomponent values that have low hemoglobin absorption among the pixelvalues of the pixels and are closest to the shape of a mucosal surfacelayer, and creates an R-component image.

Note that at this step S4312, a gradient direction may be acquired froman image created by other techniques, provided that the image has only asmall influence on the microstructures of villi on the mucosa. Forexample, an image obtained by performing an opening process by grayscalemorphology which is described in the process of the convex shapehigh-frequency component computing unit 121 may be used.

Subsequently, the gradient direction computing unit 311 b computes, fromthe R-component image, a gradient direction for the position of the arcwhose curvature central direction is computed by the curvature centraldirection computing unit 311 a. Specifically, a first derivative filterprocess (a Prewitt filter, a Sobel filter, etc.) which is a filteringprocess for computing edge strength (reference: Computer Graphic ArtsSociety, “Digital Image Processing”, pp. 114-117) is performed.

At step S4313, the direction feature data computing unit 311 outputs thecurvature central direction of the arc and the gradient direction asfeature data. Thereafter, processing returns to the main routine.

At step S432 subsequent to step S431, the frequency distributioncreating unit 312 creates a frequency distribution of the feature dataoutputted from the direction feature data computing unit 311.Specifically, a frequency distribution of the curvature centraldirections of the arcs is created for each arc gradient direction.

At step S433, the intra-region curvature feature data computing unit 310outputs the frequency distribution of the curvature central directionsof the arcs as curvature feature data.

Next, processes performed by the abnormality determining unit 320 willbe described.

The abnormality determining unit 320 determines whether there is anabnormal portion in the region of interest, based on the frequencydistribution of curvature central directions which is curvature featuredata.

Here, if there are abnormal villi in the region of interest and whenimaging is performed on the inclination of an object (mucosal structure)in one imaging direction as illustrated in FIG. 32A, a linear image inwhich linear regions PR6 face substantially in one direction is obtainedas illustrated in FIG. 32B. In this case, as illustrated in FIG. 32C,curvature central directions d face in a nearly uniform direction.Therefore, as illustrated in FIG. 32D, bias occurs in a distribution ofcurvature central directions for each gradient direction.

On the other hand, if there is a bubble region in the region of interestand when imaging is performed on the inclination of an object (mucosalstructure) in one imaging direction as illustrated in FIG. 33A, a linearimage is obtained in which linear regions PR7 have a circular shapecorresponding to the contour of the bubble region, as illustrated inFIG. 33B. In this case, as illustrated in FIG. 33C, curvature centraldirections d face in all directions. Therefore, as illustrated in FIG.33D, the curvature central directions for each gradient direction aredistributed widely and nearly uniformly.

Hence, when there is a bias in curvature central direction in thefrequency distribution of curvature central directions created for eachgradient direction, i.e., the variance of curvature central directionsis less than or equal to a predetermined threshold value, theabnormality determining unit 320 determines that there is an abnormalportion in the region of interest.

As described above, according to the third embodiment, a mucosal regionwhich is extracted from an intraluminal image is set as a region ofinterest, and high-frequency components included in convex regions, eachof which is composed of a pixel group having higher pixel values thanits neighboring pixels, are extracted from the region of interest. Then,for linear regions obtained by thinning the high-frequency components,curvature central directions of arcs along the linear regions andgradient directions are computed, and it is determined whether there isan abnormal portion in the region of interest, based on a frequencydistribution of the curvature central directions created for eachgradient direction. Accordingly, abnormal villi and bubble regions areaccurately identified according to the orientations of villi that changedepending on the imaging direction, enabling to improve the accuracy ofdetermination of an abnormal portion.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be described.

FIG. 34 is a block diagram illustrating a configuration of an imageprocessing apparatus according to the fourth embodiment. As illustratedin FIG. 34, an image processing apparatus 4 according to the fourthembodiment includes a calculating unit 400 instead of the calculatingunit 100 illustrated in FIG. 1. The calculating unit 400 includes aregion-of-interest setting unit 410, a linear convex region extractingunit 120, an intra-region curvature feature data computing unit 420, andan abnormality determining unit 430. Note that the configuration andoperation of the linear convex region extracting unit 120 are the sameas those of the first embodiment. Note also that the configurations andoperation of those units of the image processing apparatus 4 other thanthe calculating unit 400 are also the same as those of the firstembodiment.

The region-of-interest setting unit 410 includes a mucosal regionextracting unit 111 that extracts a mucosal region by excluding regionsother than mucosa, such as residues and dark portions, from a processingtarget intraluminal image; and a region dividing unit 411 that furtherdivides the extracted mucosal region into a plurality of regions, andsets each of the divided regions as a region of interest. Note that theoperation of the mucosal region extracting unit 111 is the same as thatof the first embodiment.

The intra-region curvature feature data computing unit 420 includes asize feature data computing unit 421 including a curvature computingunit 131 a and a curvature representative value computing unit 131 b;and a frequency distribution creating unit 132. Note that the operationof the curvature computing unit 131 a, the curvature representativevalue computing unit 131 b, and the frequency distribution creating unit132 is the same as that of the first embodiment.

The abnormality determining unit 430 determines whether there is anabnormal portion in each region of interest, based on curvature featuredata computed by the intra-region curvature feature data computing unit420.

Next, the operation of the image processing apparatus 4 will bedescribed.

The overall operation of the image processing apparatus 4 is similar tothat illustrated in FIG. 2, but the contents of processes performed atsteps S20 to S60 are different from those of the first embodiment.

FIG. 35 is a flowchart illustrating processes performed by theregion-of-interest setting unit 410 at step S20. Note that steps S201 toS205 in FIG. 35 correspond to those in FIG. 3.

At step S241 subsequent to step S205, the region-of-interest settingunit 410 divides a mucosal region remaining after removing unnecessaryregions, into a plurality of regions, each having a predetermined sizeor less. A division method is not particularly limited, and in thefourth embodiment the mucosal region is divided into rectangularregions. In addition, the size of one divided region is preset such thatthe difference in imaging distance to an object in the divided regionfalls within a predetermined range.

At subsequent step S242, the region-of-interest setting unit 410 outputscoordinate information of each of the divided regions as a region ofinterest.

At step S30, the linear convex region extracting unit 120 extractslinear regions from each of the regions of interest outputted at stepS20.

At step S40, the intra-region curvature feature data computing unit 420computes, for each region of interest outputted at step S20, curvaturefeature data based on curvatures of arcs along the linear regions. Notethat as the curvature feature data, a frequency distribution ofrepresentative values of curvatures of one or more arcs which arecomputed from each section of the linear regions is computed similarlyto the first embodiment, but unlike the first embodiment, the distanceinformation is not computed.

At step S50, the abnormality determining unit 430 determines, for eachregion of interest, whether there is an abnormal portion in the regionof interest, based on the frequency distribution of representativevalues of curvatures of arcs which is curvature feature data.Specifically, first, when the curvatures are smaller than apredetermined threshold value (curvature threshold value), mucosalcontours are displayed in the region of interest and thus it isdetermined that there is no abnormal portion.

In addition, when the curvatures are greater than the curvaturethreshold value and the variance of curvature is greater than apredetermined threshold value (variance threshold value), a bubbleregion is displayed in the region of interest and thus the abnormalitydetermining unit 430 determines that there is no abnormal portion. Thisis because even if the imaging distance is uniform, a bubble regionessentially includes bubbles with various curvatures.

On the other hand, when the curvatures are greater than the curvaturethreshold value and the variance of curvature is less than or equal tothe predetermined threshold value (variance threshold value), abnormalvilli are displayed in the region of interest and thus the abnormalitydetermining unit 430 determines that there is an abnormal portion. Thisis because essentially, abnormal villi in a neighboring region haveshapes similar to each other, and thus, when the size of a region ofinterest is small and the difference in imaging distance in one regionof interest can be ignored, the curvatures of arcs corresponding to thecontours of abnormal villi match each other.

At step S60, the calculating unit 400 outputs results of thedetermination made for the respective regions of interest at step S50,together with the coordinate information of the regions of interest.Accordingly, a control unit 10 displays the results of the determinationon a display unit 40, and records the results of the determination in arecording unit 50 so as to be associated with image data of theprocessing target intraluminal image. At this time, the control unit 10may display a mark or the like that indicates the position of a regionof interest having been determined to have an abnormal portion, suchthat the mark or the like is superimposed on the intraluminal imagedisplayed on the display unit 40.

As described above, according to the fourth embodiment, a plurality ofregions of interest are set by dividing a mucosal region extracted froman intraluminal image, and high-frequency components included in convexregions, each of which is composed of a pixel group having higher pixelvalues than its neighboring pixels, are extracted from each region ofinterest. Then, for linear regions obtained by thinning thehigh-frequency components, a frequency distribution of curvatures iscomputed as curvature feature data, and an object displayed in theregion of interest is determined based on the feature data. Accordingly,abnormal villi and bubble regions and mucosal contours are accuratelyidentified, enabling to determine whether there is an abnormal portion.

In addition, according to the fourth embodiment, the mucosal regionextracted from the intraluminal image is divided into regions of a sizeat which the difference in imaging distance to an object can be ignored,and each of the divided regions is set as a region of interest, andthen, each process for determining whether there is an abnormal portionis performed. Thus, a distance information computation process and anabnormal portion determination process according to distance informationcan be omitted, enabling to simplify a calculation process.

In addition, according to the fourth embodiment, since a determinationas to whether there is an abnormal portion is made for each region ofinterest, a region having an abnormal portion in the intraluminal imagecan be identified.

Modification 4-1

Next, a modification 4-1 of the fourth embodiment will be described.

The size of a region of interest to be set by the region-of-interestsetting unit 410 may be variable. The operation of the image processingapparatus 4 performed when the size of a region of interest is variablewill be described below. FIG. 36 is a flowchart illustrating theoperation of the image processing apparatus 4 of modification 4-1.

At step S20, the region-of-interest setting unit 410 sets a mucosalregion remaining after removing unnecessary regions, as one region ofinterest.

The contents of processes at subsequent steps S30 to S50 are the same asthose of the fourth embodiment.

At step S71 subsequent to step S50, the calculating unit 400 determineswhether a determination result indicating that there is an abnormalportion in any part of the set region of interest has been obtained.Note that in the first loop of the flowchart, the number of regions ofinterest is one.

If a determination result indicating that there is an abnormal portionhas been obtained (step S71: Yes), processing proceeds to step S60. Notethat the content of the process at step S60 is the same as that of thefourth embodiment.

On the other hand, if a determination result indicating that there is anabnormal portion has not been obtained (step S71: No), the calculatingunit 400 determines whether the size of the currently set region ofinterest is less than or equal to a predetermined size (step S72). Notethat the predetermined size at this time is set to a size at which thedifference in imaging distance to an object in the region of interestcan be ignored.

If the size of the region of interest is less than or equal to thepredetermined size (step S72: Yes), processing proceeds to step S60.

On the other hand, if the size of the region of interest is greater thanthe predetermined size (step S72: No), the region-of-interest settingunit 410 reduces the size of the currently set region of interest andre-sets the region of interest in a mucosal region (step S73).Thereafter, processing proceeds to step S30.

As described above, according to this modification 4-1, while the sizeof a region of interest is gradually reduced, a determination process asto whether there is an abnormal portion is performed for each region ofinterest. Here, by reducing the size of a region of interest, thedifference in imaging distance to an object in each region of interestis reduced. Thus, the accuracy of determination of an abnormal portionfor each region of interest increases by repeating the loop of steps S30to S73. By thus changing the accuracy of determination of an abnormalportion in an intraluminal image from low to high, overlooking ofdetermination of an abnormal portion can be suppressed while an abnormalportion determination process is made efficient.

The image processing apparatuses according to the above-described firstto fourth embodiments and modifications thereof can be implemented byexecuting an image processing program recorded in a recording device, bya computer system such as a personal computer or a workstation. Inaddition, such a computer system may be used connected to other computersystems or devices such as a server, through a local area network, awide area network (LAN/WAN), or a public line such as the Internet. Inthis case, the image processing apparatuses according to the first tofourth embodiments may acquire image data of an intraluminal imagethrough these networks, or output image processing results to variousoutput devices (a viewer, a printer, etc.) which are connected throughthese networks, or store image processing results in storage devices(e.g., a recording device and a reading device therefor) which areconnected through these networks.

According to some embodiments, a linear region having a predeterminednumber or more of continuously-arranged pixels whose pixel values arehigher than those of their neighboring pixels is extracted, curvaturefeature data based on the curvatures of one or more arcs along thelinear region is computed, and it is determined whether there areabnormal villi, based on a distribution of the curvature feature data.With this feature, even if one image includes regions with differentimaging distances to an object, it is possible to accurately determinewhether there are abnormal villi.

Note that the present invention is not limited to the first to fourthembodiments and the modifications thereof, and various inventions can beformed by appropriately combining together a plurality of componentswhich are disclosed in each embodiment or modification. For example, aninvention may be formed by excluding some components from all componentsshown in each embodiment or modification, or an invention may be formedby appropriately combining together components shown in differentembodiments or modifications.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

What is claimed is:
 1. An image processing apparatus comprising: aregion-of-interest setting unit configured to set a region of interestin an image; a linear convex region extracting unit configured toextract, from the region of interest, a linear region having apredetermined number or more of continuously-arranged pixels whose pixelvalues are higher than pixel values of neighboring pixels; anintra-region curvature feature data computing unit configured to computecurvature feature data based on curvatures of one or more arcs along thelinear region; and an abnormality determining unit configured todetermine whether there is an abnormal portion in the region ofinterest, based on a distribution of the curvature feature data.
 2. Theimage processing apparatus according to claim 1, wherein theintra-region curvature feature data computing unit comprises a sizefeature data computing unit configured to compute distance informationand the curvatures of one or more arcs, and the abnormality determiningunit is configured to determine that there is an abnormal portion in theregion of interest when the distribution of the curvatures is within arange smaller than a predetermined threshold value, the predeterminedthreshold value being determined according to the distance information.3. The image processing apparatus according to claim 2, wherein the sizefeature data computing unit comprises: a curvature computing unitconfigured to compute the curvatures of one or more arcs from each ofsections of the linear region, the sections being delimited by endpointsof the linear region and/or intersection points of linear regions; acurvature representative value computing unit configured to compute arepresentative value from the curvatures of one or more arcs; and adistance information computing unit configured to compute the distanceinformation from an imaging position of the image to the linear region.4. The image processing apparatus according to claim 1, wherein theintra-region curvature feature data computing unit comprises a shapefeature data computing unit configured to compute a variation in thecurvatures of one or more arcs, and when the variation is greater than apredetermined value, the abnormality determining unit is configured todetermine that there is an abnormal portion in the region of interest.5. The image processing apparatus according to claim 4, wherein theshape feature data computing unit comprises: a curvature computing unitconfigured to compute the curvatures of one or more arcs along thelinear region from each of sections of the linear region, the sectionsbeing delimited by endpoints of the linear region and/or intersectionpoints of linear regions; and a curvature standard deviation computingunit configured to compute, for each of the sections, a standarddeviation of the curvatures of one or more arcs, wherein the abnormalitydetermining unit is configured to make a determination based on thestandard deviation.
 6. The image processing apparatus according to claim1, wherein the intra-region curvature feature data computing unitcomprises a gradient direction feature data computing unit configured tocompute directions going toward a curvature center from each of the oneor more arcs, and directions in which an object is inclined in a depthdirection of the image, at positions of each of the one or more arcs,and when a variance of frequency of the directions going toward thecurvature center is less than or equal to a predetermined thresholdvalue, the abnormality determining unit is configured to determine thatthere is an abnormal portion in the region of interest, the variance offrequency being created for each of the directions in which the objectis inclined.
 7. The image processing apparatus according to claim 1,wherein when it is determined that there is no abnormal portion in theset region of interest, the region-of-interest setting unit isconfigured to reduce a size of the region of interest to re-set theregion of interest.
 8. An image processing method comprising: setting aregion of interest in an image; extracting, from the region of interest,a linear region having a predetermined number or more ofcontinuously-arranged pixels whose pixel values are higher than pixelvalues of neighboring pixels; computing curvature feature data based oncurvatures of one or more arcs along the linear region; and determiningwhether there is an abnormal portion in the region of interest, based ona distribution of the curvature feature data.
 9. A non-transitorycomputer-readable recording medium with an executable program storedthereon, the program instructing a processor to execute: setting aregion of interest in an image; extracting, from the region of interest,a linear region having a predetermined number or more ofcontinuously-arranged pixels whose pixel values are higher than pixelvalues of neighboring pixels; computing curvature feature data based oncurvatures of one or more arcs along the linear region; and determiningwhether there is an abnormal portion in the region of interest, based ona distribution of the curvature feature data.